Blog ala Vidar

SQL, AppFrame and other cool technologies

Category Archives: Published

4. Nov; Mission (Almost) Impossible

The mission of the day was to find the TechEd area, which seemed like an impossible task for a couple of hours.

People from Barcelona are not too good in speaking English, and when the city is this big, and we only got a name of a building, it seemed to be very hard to explain where to go. Even the bus-drivers didn’t know where CCIB (Centre Convencions Internacional Barcelona) were.

After walking down the (too long) street, La Rambla, we desided to start asking people where this was. People were pointing in all directions….

With 18 degrees celcius, this wasn’t too bad. We got to taste Spanish (I believe they were Belgish!) wafles with caramel! 😀

After a visit to a big aquarium, we desided to grab a taxi, which eventually got us to CCIB.

I figured out, bringing me to TechEd is just like bringing a kid to a candy factory, or a muslim to Mekka! Next to heaven!

 

When we registered, we got a TechEd Developers 2007 computer bag with ALOT of cool things in it! Among these things were:

Several magazines, Conference Schedule, Developers editions of mobile dev tools, VS, pen, and a hole lot of advertisement from the sponsors.

There were nothing else this day, other than registration which took about 2 minutes.

 

After coming back to the hotel, after at least 7000km of walking (or at least, my feet felt so), we went to a great Tapas-place and tried the “Tasting menu” with 7 dishes, and “a couple” of beers. (PS! When you order cognac, you do not get cognac!)

3. Nov; Arrival

Saturday, 3rd nov my day started at about 10am. Packing my bag, passport and credit-cards!

From Stavanger we went to Amsterdam (Schiphol). The VIP Lounge (for Diners-cardholders) is a great place, except when there’s a lady behind the desk at about 120 years old. I don’t think the word “service” exists in her dictionary.

Anyhow, from Amsterdam to Barcelona, and from the airport with a taxi-driver who only understood “beer” and “monetas”, we understood getting around town could be difficult! We ended up at the hotel late in the evening, had “a couple” of beers and got to bed.

We’re living at Hotel Catalonia Ramblas, which by the way is in the longest street (La Ramblas) in the Europe.

 

“We” are:

Vidar, Torgeir, Jan Leon, Øystein, Nils Arne, Ronny and his two little helpers; Beate and Sidsel

Omega’s security conecpt

Wouldn’t it be a great world if everyone had access to everything, but only used the things they should be using? Well, unfortunately humans do things they are not supposed to do. Either it’s because of curiosity, or stupidity. Because of this we have a security concept in Omega built on both SQL server security and our own AppFrame Security. Today I’m going to go into details about how this works.

We separate the security into two sections, SQL and AppFrame. First, let’s have a look at how SQL security is built up.

SQL Security

Login – The account you are using when authenticating to the server
User – The account in the database which is linked up to your login, to specify which databases you’ve got access to.
Roles – A database “group” which is used for adding permissions. Instead of adding permissions to ALL (maybe a couple of thousand) users, you add them to a role and add permissions to this role.

We got one role, called af_user. All users should be a member of this role. This role got “SELECT *” permissions to all views, EXECUTE permissions on stored procedures and functions. When it comes to tables, we have a special concept here. We give af_user only access to the PrimKey and Timestamp fields on the tables. This is because PrimKey is the PRIMARY KEY on the table. Timestamp is the row-version field, so the application knows if someone else has updated the row since you last picked it up. af_user also got full INSERT, UPDATE and DELETE permissions to the tables. This is actually why we are using views (tbv, tbx). The user got full access to the view. When he’s trying to update data from the view, it will update the table. When access tries to update (or delete) a record, it uses a property called “unique table”. Here you specify which table should be updated. So, access actually uses the table, and not the view, when running updates (delete, insert etc). Therefore, the user needs access to SELECT PrimKey and Timestamp in the table.

Example:

Recordsource: SELECT * FROM stbv_System_Groups

You try to change the Description of the Administrators group in the Omega Domain. Access will then run something like this query to the server:

UPDATE stbl_System_Groups
    SET Description='Immortal users'
    WHERE PrimKey='C35F02E6-BA01-4DFD-A697-D4A2A6B8744A'
      AND Timestamp='0x000000000BB2BAC5'

Notice it is using stbl, and not stbv, even though you’re using stbv in the recordsource. This is because access reads the view and sets the unique table property itself if it’s not set by the developer.

AppFrame Security

In AppFrame we got both MasterGroups and Groups. The difference between these is that Groups are domain dependant. MasterGroup is not. So, what is a domain? Well, it’s used to separate data into areas where they belong. For example, in Statoil they use one domain for each project. Snohvit got one domain, OrmenLange got one domain, KEP2010 got one etc. This is because the people working with Snohvit doesn’t care about what’s happening at Langeled. So, therefore we got Groups which are domain dependant. But, we also got (Master) Groups which are not domain dependant. Typical groups like Administrators, CMS RO and CMS RW. Master Groups is like a template for Groups. Take “Administrators” as an example. We add all permissions (I’ll come back to this subject) we need as administrator (mainly SUID on all tables), and then assign it to domains. What actually happens when you assign a Master Group to domains is that you make a new group in that domain, based on the Master Groups permissions. If a user is member of the Master Group, he will automatically be a member of the Groups which “belongs” to the Master Group. If you want one user to not be an Administrator in all domains, only Snohvit, you add him to the Group in the Snohvit domain, and not in the Master Group. Smart? Well, be careful! If you delete a member of the Master Group, you will also delete that member from the Groups which has this Master Groups as a parent.

You might have noticed, we use both tbv and tbx as table views.

Tbv’s are used to get data from a table in your current domain.
Tbx’ are used to get data from a table in ALL domains you’ve got access to.

Table Rows Permissions

In each Group we got a set of Table Rows Permissions. This table contains the following fields:

Domain – to specify which domain this permission set belongs to.
GroupRef – The PrimKey for the group this permission set belongs to.
TableID – The table name this permission set belongs to.
PGrant – Which permissions the group got on this table. S for SELECT, U for UPDATE, I for INSERT and D for DELETE. Min: S, max: SUID
Criteria1, Criteria2, Criteria1Revoke, Criteria2Revoke – I’ll get back to these later.
MasterTRRef – Defines if this record has a Master Table Row. If it does, this group is controlled by a Master Group.

All our tbv’s should look something like this:

SELECT *
	FROM dbo.stbl_System_Groups (NOLOCK)
	WHERE EXISTS (SELECT *
			FROM sviw_System_MyPermissionsCurrentDomain (NOLOCK)
			WHERE ISNULL(Domain, '') = stbl_System_Groups.Domain
			  AND TableID = 'stbl_System_Groups')

sviw_System_MyPermissions_CurrentDomain will return ALL your permissions in your current domain. Your current domain is saved in the field CurrentDomain in stbl_System_Users. This query will look for records with TableID = ‘stbl_System_Groups’ and make sure it only return records where stbl_System_Groups.Domain is like your current domain. This is how the SELECT works.

Triggers

Now, let’s move on to UPDATE, INSERT and DELETE. These actions are handled by triggers. stbl_System_Groups_ITrig is the INSERT trigger for this table. In this trigger you’ll find:

IF @@RowCount = 0 RETURN
IF IS_MEMBER('db_owner') = 1 GOTO ContinueTransaction

IF EXISTS (SELECT MyP.*
		FROM dbo.sviw_System_MyPermissions MyP
		JOIN Inserted
			ON ISNULL(MyP.Domain, '') = ISNULL(Inserted.Domain, '')
		WHERE MyP.TableID='stbl_System_Groups'
		  AND MyP.PGrant LIKE '%I%') GOTO ContinueTransaction

RollbackTransaction:
    RAISERROR ('---- No insert permissions on table: stbl_System_Groups ----',18,1) ROLLBACK TRANSACTION
    RETURN

ContinueTransaction:
    SET NOCOUNT ON

What happens here is that it first checks if there’s any records “thrown” at the trigger. If not, it quits the trigger. Then it checks if you are a member of the builtin db_owner database role. If you are, you should have access to the table, so no need in checking AppFrame permissions. If you are not a member of this role, it will continue, to check if you got records with INSERT permissions on this table. If you do, you continue to ContinueTransaction, if not, you get an error saying “—- No insert permissions on table: stbl_System_Groups —-“. 18 is the severity level of this message. 18 is the highest normal (mortal) users are allowed to use. This will cause an MsgBox to show in access. If this message is raised, the trigger will rollback all changes.

DBTables

I said I would come back to Criteria1, Criteria2 etc later. Now it is “later”. Till now I’ve only told you about table level security. Now, let’s discuss ROW level security. As a part of AppFrame Security, you’ve got the privilege of using row level security. This is actually one of the main reasons we’re not only using SQL security. MSSQL are not supporting row level security. They said they would in 2005, then they said they would in 2008. Still no support for it though! In other words, we got something more fancy than Microsoft 🙂 Anyhow. This means that you can specify WHICH rows you want a user to be able to see, add, update or remove by the use of a filter. This is actually what Criteria1 and Criteria2 are used for. Here you specify the filter. Which fields to filter on is desided in stbl_System_DBTables. This table contains the following fields (I’ve removed the fields you do not need to worry about):

  • ObjectName – The table name
  • CField1 – The first field to have a filter on if using row level security. This uses LIKE
  • CField2 – The second field to have a filter on if using row level security. This uses LIKE
  • CField1Revoke – (bit) Set this to 1 if you’d like CField1 to use “NOT LIKE” instead of “LIKE”.
  • CField2Revoke – (bit) Set this to 1 if you’d like CField2 to use “NOT LIKE” instead of “LIKE”.
  • PredecessorTable – If you have parent-child tables, your child table may have its parent table in this field. This way the child table will inherit the parent’s permissions.
  • NoDataLog – (bit) Skip INSERT to stbl_System_DataLog in the triggers
  • ExtendedDataLogging – (bit) If this is set, the trigger will log every change to all fields in stbl_System_DataLog
  • StandardSecurity – (bit) This table has tbv (and tbx) and standard triggers.
  • BypassSecurityForDBO – (bit) This adds the following line to the triggers: IF IS_MEMBER(‘db_owner’) = 1 GOTO ContinueTransaction
  • DenyafUserUpdatePermissions – (bit) This makes ApplyPermissions REMOVE UPDATE, INSERT and DELETE permissions from af_user on the table. Do NOT use this unless you’re only using SPs on this table!
  • GrantAccessForClusteredIndex – (bit) This makes ApplyPermissions also add SELECT permissions on all fields that are in the clustered index on the table, in addition to PrimKey and Timestamp. The reason is that when you use Access on SQL 2005, it will use the clustered index, instead of the primary key, for SELECT when updating.

To change the settings in DBTables, open Query Analyzer, rightclick the table you want to change and choose “Security (Views and Triggers)”

Tuning SQL code

Have you ever though of tuning your car? Yeah, why not?
It makes your car run faster and use less fuel. Well, for starters it’s a bit expensive. But, tuning SQL is free, and it’s even more fun than tuning your car. Tuning your SQL means your queries run faster and use less fuel (CPU, memory etc)!

I’m at Conocophillips Indonesia at the moment upgrading to SQL Server 2005. I’ve used a couple of days to tune different queries, and at the most I got a query to run from an average of 4 minutes, to about 10 seconds, with basically the same code! Do NOT tell me that this was a temporary solution, because my solution was even easier! 🙂

When writing SQL code, keep these simple rules in mind:
Do not use sub-queries unless it’s ABSOLUTELY necessary
If you are setting several variables from the values of fields in tables/views, use one SELECT statement instead of three SET statements.

For example, do not use:

SET @Variable1 = 'Test'
SET @Variable2 = (SELECT Test FROM Test1 WHERE Test3='asdf')
SET @Variable3 = (SELECT Test2 FROM Test1 WHERE Test3='asdf')

Use:

SELECT TOP 1
  @Variable1 = 'Test',
  @Variable2 = Test,
  @Variable3 = Test2
    FROM Test1
    WHERE Test3 = 'asdf'

You can also use SELECT instead of SET to set static values
If possible, do not use CASE or UNION (ALL)
Use EXISTS, instead of 0 < (SELECT COUNT(*) something)
Use TOP 1. Always remember that a sub-query MAY return more than one record. Your query might then fail if you’re using Field = (SELECT SubField FROM subtable).

These rules are things that every developer should keep in mind when writing SQL code.

Of course, there might be places where you’ve followed every rule, but it still takes minutes to run. Then you might want to see the execution-plan. There are a couple of major things too look for here. For example, Scans (Index Scan, Clustered Index Scan, Table Scan etc) is NOT good. Seeks are a good thing.

You might also want to see what’s really happening when you’re running your query. To turn on execution-plan in Server Management Studio, press ctrl + M and rerun your query. It will appear as a new tab beside Results and Messages.

And, as always; Profiler might be a good tool. If you let it run for a couple of hours you will discover queries which take longer time than others. For example, you can set a filter on the Reads field, for >= 50000.

Do you have any other simple roules you’d like to share with us? Please mail them to me, and I’ll publish them here!

Katmai is available!

For those of you who thinks Katmai is a moutain, well.. I am not sure.

But, I do know that it’s the code name for the future version of SQL Server from Microsoft. They call it: Microsoft SQL Server 2008, “Katmai” June CTP

After reading the release notes from MS, I was very disapointed, but after testing it I got new hope. The first thing I layed my eyes on was something called “Declarative Management Framework” (DMF). This includes something I think we might use in the future: policies. I see great potential with these policies because you can setup naming conventions on the server. For example, you can deny saving tables with names that does not start with [as][tbl], or you can force login-names to start with a prefix (s for statoil, k for kellogg). With policies you can also set how much CPU a user can use.

To enable these options, I had to run:

C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\BIN\sn.exe -Vr Microsoft.SqlServer.DmfSqlClrWrapper,89845DCD8080CC91

and then:

ALTER DATABASE [databasename] SET TRUSTWORTHY ON

Intellisence

I have tried this feature from Red Gate’s ”SQL Prompt”. To be honest, I am not a fan of this, because it is slow, it always gets in my way and I like writing the code myself. (used to vim and notepad ). BUT, I do understand that this is a very neat feature for others. Microsoft sais the only reason it didn’t come with this CTP is that they need to make it run faster.

Storing files

It should now be possible to use a mix of storing files to a fileshare and a blob-field. I have not found the details about this, but it is something about making an own filegroup (called FILESTREAM) and in this way combining fileshare and blob-fields.

Compability

For me, it seems like this should rather be called SQL Server 2005 SP1, than SQL Server 2008. In my opinion, there is not enough changes to make a new version. Anyhow. Omega already supports 2005, and I have tested an restore of R2CurrentBuild and we are compatible with the upcoming 2008 too!

PS! SQL Server 2008 compared to 2005, seems much like .NET 3.0 (3.5) compared to .NET 2.0.

Indenting

Have any of you ever had problems with understanding a query because of the (lack of) indenting? I’ll bet the answer is “Yes, MANY times!”.

Today, as you’ve probably already understood, I’m going to take a look at how easy indenting is, and how much time it saves you, me, Bill Gates and all other poor creatures. Some may call me strict when it comes to indenting, but I think it’s a good thing. I’m not saying everyone should follow my standard. I just want to spread the word and make people understand.

We can start with an example from an old view in AppFrameR2 (aviw):

Before:

select d.title as "teamdoctitle", e.*,
(select count(*) from stbv_calendar_events e2 where e2.copiedfromeventref=e.copiedfromeventref and e.copiedfromeventref is not null) copycount,
cast(case when exists(select * from stbl_calendar_attendance where login = suser_sname() and eventref = e.primkey) then 1 else 0 end as bit) as userattend
from stbv_calendar_events e (nolock) left outer join stbv_teamdoc_documents d (nolock) on d.primkey=e.teamdocref where e.primkey = @primkey

After:

SELECT D.Title AS TeamDocTitle, E.*,
  (SELECT COUNT(*)
        FROM stbv_Calendar_Events AS E2
        WHERE E2.CopiedFromEventRef=E.CopiedFromEventRef
          AND E.CopiedFromEventRef IS NOT NULL) CopyCount,
  CAST(CASE WHEN EXISTS(SELECT *
                            FROM stbl_Calendar_Attendance
                            WHERE Login = SUSER_SNAME()
                              AND EventRef = E.PrimKey)
                    THEN 1
               ELSE 0
               END AS BIT) AS UserAttend
    FROM stbv_Calendar_Events AS E WITH (NOLOCK)
    LEFT OUTER JOIN stbv_TeamDoc_Documents AS D WITH (NOLOCK) ON D.PrimKey = E.TeamDocRef
    WHERE E.primkey = @PrimKey

As you see, it’s difficult to see what’s SQL and what’s not. It is also very hard to know where the subqueries end.

Imagine you have a MUCH longer query. Sviw_MSAccessCR2_MyMenuObjects for instance. In this view you have alot of case, subqueries (with joins) and both inner and outer joins. Try reading that without indenting, syntax highlight and uppercasing SQL-commands! That’s not even possible! About a year ago we had some performance issues with this view. I remember my thoughts:
– ”EHM! Where do I start?!”

It was about 13 kilometers longer than the one today, it had alot of unions and other ”fancy” stuff. I think I used almost a day only ”getting to know” (indenting/casing) the view! If possible, take a look at it, remove all indenting and change all text to lowercase (mark it and press ctrl + shift + L). Copy it to notepad and try to figure out where the subqueries end, where each case ends etc. It’s almost impossible.

Very often I get questions like ”Could you help me with a SQL-query?” on mail and/or MSN. To understand the logic, I always start with indenting (if it’s not already indented). I’ll bet $100 that most of these questions wouldn’t be asked if the person already had indented the query, because it’s so much easier to find the issue with, than without.

It takes me more time to write a query WITHOUT indenting and casing than WITH, because I’m so used to this method

.. and it’s MUCH more difficult to read. My advice: Use indenting and uppercase on SQL-commands. If it’s ”my” standard or some improved (if that’s possible 😉 standard does not matter. ANYTHING is easier to read than one-line-plain-sql-queries!

24th

Chit-chat:

We just had “Pizza & Learning” here in Ølen. My opinion: very successful 🙂 First, Peter startet with showing all available functionality in the grid-control. After half an hour, came FINALLY the pizza-guy. 14(?) HUNGRY computer-geeks to feed!

After some pizza and chit-chat, I started trasking about the new features in SQL 2005, compared to 2000. This includes a couple of new functions and fields (ROW_NUMBER, RANK, VARCHAR(MAX) etc).

But, wait. THERE’S MORE!

Something that can be, in my opinion, VERY useful for developers is the new Dynamic Management Views and Functions (DMV / DMF). This is internal views, which gives you ALOT of (un)usefull information about queries, indexes etc. My example was the following:

SELECT * FROM sys.dm_db_index_physical_stats (5, 665678065, NULL , NULL, 'LIMITED')

5 = DB_ID(‘AppFrameR2’)
665678065 = OBJECT_ID(‘stbl_System_Users’)

This returns everything worth knowing about the indexes on stbl_System_Users. What you should notice here is the “avg_fragmentation_in_percent” column. If this is high, you might want to rebuild your index. This is why I have made a new SP: sstp_System_RebuildIndexes

First time I ran that on AppFrameR2 it took somewhere between 3 and 5 hours (goldfish memory). Now it only takes 30-45 minutes, and I’ve run it each weekend since the first time i ran it. OBS!! I do NOT recommend running this within workhours! This will slow down, and sometimes LOCK several tables! Therefore: Run it at night, when everyone SHOULD be in their bed.

Torgeir’s answer to everything:
One other thing, that’s also available in 2000: PROFILER!

Each time I ask Torgeir (Omegas tech. guru number UNO?) a question related to databases, he ALWAYS (I’m not kidding!) answers: Have you tried the profiler?
… and each time I get that answer, I get red like a tomato and mumble his answer to myself. 90% of the time, this helps, the other times, I ask again, and get a “real” answer 😛

Profiler can help you with alot of things. SQL “debuging” of queries, what happens in stored procedures, checking WHAT’S taking so long and alot of other cool things. If you haven’t tried it: WHY NOT? PS! If anyone’s wondering if I got a “What would Torgeir do” t-shirt: No, actually not. The norwegian mail-service has lost it!

Arild’s cute little query:

SELECT TOP 5 total_worker_time/execution_count 'Avg CPU Time',
  SUBSTRING(st.text, (qs.statement_start_offset/2)+1,
  ((CASE qs.statement_end_offset WHEN -1 THEN DATALENGTH(st.text)
    ELSE qs.statement_end_offset END - qs.statement_start_offset)/2) + 1)statement_text
        FROM sys.dm_exec_query_stats AS qs
        CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS st
        ORDER BY total_worker_time/execution_count DESC

You might want to try this cute little query, just for fun. Or you might end up with tuning the result. Who knows 😉
Anyways, open your copy of books online, and in the “Look for:” textbox, write: sys.dm_db
You’ll get a list of dynamic management views and functions. This is excellent reading material on a saturday night, with a glass, or maybe a bottle?, of wine 🙂